Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity

Abstract This paper analyzes three formal models of Transformer encoders that differ in the form their self-attention mechanism: unique hard attention (UHAT); generalized (GUHAT), which generalizes UHAT; and averaging (AHAT). We show UHAT GUHAT Transformers, viewed as string acceptors, can only recognize languages complexity class AC0, recognizable by families Boolean circuits constant depth polynomial size. upper bound subsumes Hahn’s (2020) results cannot DYCK or PARITY language, since those are outside AC0 (Furst et al., 1984). In contrast, non-AC0 MAJORITY DYCK-1 AHAT networks, implying cannot.